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[57] ABSTRACT 

An adaptive method for removing fixed pattern noise from 
focal plane array (FPA) imagery. A set of correction terms is 
applied to the focused image from the FPA, and a filter is 
applied to the corrected, focused image. The set of correc- 
tion terms is also applied to a blurred version of the FPA 
image, and the filter is applied to the corrected, blurred 
image. Fixed pattern noise errors are then calculated using 
the filtered imagery, and employed to update the correction 
terms. The updated correction terms are then used for 
processing the next image. In one embodiment, the filter is 
an anti-median filter. In another embodiment, the filter is an 
anti-mean filter. 
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ADAPTIVE NON-UNIFORMITY 
COMPENSATION ALGORITHM 

TECHNICAL FIELD OF THE INVENTION 

This invention relates to processes for the removal of 5 
fixed pattern noise from sensor images. 

BACKGROUND OF THE INVENTION 

Focal plane arrays (FPAs) are used in various applications 
to capture images for subsequent processing. For example, io 
air-to-air missiles have employed infrared sensors with FPAs 
to capture infrared images of a target being tracked. The 
focal plane array contains nxn infrared detectors. Each 
detector has a slightly different sensitivity to infrared radia- 
tion. This non-uniform sensitivity yields fixed pattern noise 15 
(FPN). FPN manifest itself in the image resulting in some 
pixels which are too bright and some which are too dark. 

Some missile sensor processors have employed algo- 
rithms to reduce FPN. Currently used algorithms have 
introduced significant scene artifacts in the output image 20 
which causes the missile tracker to mis-track the target. 
Efforts to reduce the scene artifact growth have resulted in 
insufficient fixed pattern noise reduction under certain simu- 
lated flight conditions. 

25 

It would therefore be advantageous to provide a technique 
for removal of fixed pattern noise from sensor images 
without introducing significant scene artifacts. 

SUMMARY OF THE INVENTION 

An adaptive method for removing fixed pattern noise 30 
from focal plane array (FPA) imagery is described. The 
method comprises the following steps: 

obtaining respective focused and blurred digitized ver- 
sions of an FPA image; ^ 
applying a set of correction terms to the focused version 
of said FPA image to provide a corrected, focused FPA 
image; 

applying a filter to the corrected, focused version of said 

FPA image to obtain a filter focus image; 40 
applying said set of correction terms to said blurred 
version of the FPA image to provide a corrected, 
blurred version of said FPA image; 
applying said filter to the corrected, blurred image to 

provide a filter blur image; 45 
calculating fixed pattern noise errors using the filter focus 

image and the filter blur image; and 
employing said fixed pattern noise errors to update the 
correction terms for use in processing a next FPA image. 

In a first embodiment, the filter includes an anti-median 50 
filter which provides, for each pixel in an image, a corre- 
sponding anti-median value which is a measure of how the 
pixel differs from a median value of the pixel and a set of 
neighboring pixels. In a second embodiment, the filter 
includes an anti-mean filter which provides, for each pixel in 55 
an image, a corresponding anti-mean value which is a 
measure of how the pixel differs from a mean value of the 
pixel and a set of neighboring pixels. 

BRIEF DESCRIPTION OF THE DRAWING 

oU 

These and other features and advantages of the present 
invention will become more apparent from the following 
detailed description of an exemplary embodiment thereof, as 
illustrated in the accompanying drawings, in which: 

FIG. 1 is a simplified schematic diagram of a missile 65 
tracking system with which the present invention is advan- 
tageously utilized. 
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FIG. 2 illustrates a sequence of images 1 . . . n+1, 
alternating between a focussed image and a blurred image, 
produced at the sensor output of the system of FIG. 1. 

FIG. 3 is a top level flow diagram illustrative of a first 
embodiment of the adaptive NUC process of the invention, 
FIG. 4 illustrates the focussed image processing for the filter 
system of FIG. 3. 

FIG. 5 is a flow diagram showing the blurred image 
processing for the filter system of FIG. 3. 

FIG. 6 illustrates the application of the anti-median filter 
processing of the filter system of FIG. 3. 

FIGS. 7A-7D illustrate four filter patterns used in the 
filter system of FIG. 3. 

FIG. 8 illustrates the cycling of the filter patterns for each 
successive pair of focus/blur image frames. 

FIG. 9 illustrates diagrammatically the updating of the 
correction terms employed in the system of FIG. 3. 

FIG. 10 illustrates diagrammatically the application of the 
correction terms to the input image for the filter system of 
FIG. 3. 

FIG. 11 illustrates an alternate embodiment of the inven- 
tion. 

FIG. 12 illustrates the focussed image processing for the 
filter system of FIG. 11. 

FIG. 13 is a flow diagram showing the blurred image 
processing for the filter system of FIG. 11. 

FIG. 14 illustrates the application of the anti-mean filter 
comprising the system of FIG. 11. 

FIGS. 15A and 15B respectively show two filter patterns 
used for the filter system of FIG. 11. 

FIG. 16 shows the cycling of the filter patterns of FIGS. 
15A and 15B for each successive pair of focus/blur frames. 

FIG. 17 diagrammatically illustrates the updating of the 
correction terms by the filter system of FIG. 11. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

FIG. 1 is a simplified schematic diagram of a missile 
tracking system with which the present invention is advan- 
tageously utilized. In general the tracking system includes a 
sensor 10 which produces an image. Typically, the sensor 10 
includes optics 12 for collecting and focusing incident 
infrared energy from the ambient scene and target on the 
focal plane array 14, which produces an infrared image of 
the scene/area being tracked. The sensor 10 includes an 
analog-to-digital (A/D) convenor 16, which converts each 
of the nxn detector outputs into digital (numeric) values or 
pixels. The digitized images are processed in accordance 
with the invention by the adaptive non-uniformity compen- 
sation (NUC) processor 20 to provide corrected output 
images 30 in which scene artifacts have been reduced, and 
fixed pattern noise has been reduced or eliminated. The 
NUC processor 20 adjusts each pixel to compensate for the 
differing sensitivity of each detector in the FPA 14. The 
missile tracking system (not shown) then employs the cor- 
rected output images in the tracking of targets. 

FIG. 2 illustrates a sequence of images 1 . . . n+1, 
alternating between a focussed image and a blurred image. 
The sensor output yields alternating focused and blurred 
images. The focus and blurring is controlled by the sensor 
optics 12. One exemplary way to achieve the focus and 
blurred images is to use a filter wheel in the sensor optics 12. 
The filter wheel spins, and has separate optics/lens to 
provide the focus image and the blurred image. The blurred 
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images are used as part of the NUC process to help dis- of images 7 and 8, filter pattern 4 is used, with the filter 

criminate scene and target from FPN. pattern cycle starting again with images 9 and 10. 

FIG. 3 is a top level flow diagram illustrative of a first The comparison function 24B2 (FIG. 5) is now described, 

embodiment of the adaptive NUC process of the invention. Each value (row, column) of the anti-median image (focus) 

The input image 18, which is either a focused image 18Aor 5 ^ compared to the corresponding value of the anti-median 

a blurred image 18B, is received from the sensor 10. The ima ge (blur) as follows. Test 1: check whether the sign (±) 

correction terms 26 are applied to the input image at step 22 of the anti-median focus is equal to the sign (±) of the 

to yield a corrected output image. The correction terms are anti-median blur. Test 2: check whether or not the ami- 

initially zeros. The corrected output image 30 is passed both median blur is greater than some fraction (say, V*) of the 

to the tracking system and to the NUC filter system 24. A 10 anti-median focus. If Test 1 and Test 2 pass, then the error 

different NUC filter is used for both the focused and blurred lerm is set equal to the anti-median focus; otherwise the 

images. These two filters are described below. The filter error term is set to zero. 

system 24 updates the correction terms 26 based on the The calculation of the anti-median can be positive or 

corrected output image. negative depending if the center pixel "X" is greater than or 

FIG. 4 illustrates the focused image processing for the 15 less , thaa the median value. For example, assume that the 

filter system 24 of FIG. 3. The input focused image 18A is median ts 100 and the center pixel is 90. The anti-median 

received from the sensor, and the correction terms 26 are would * e ? ual to 100 minus 90, or positive 10 But if the 

applied at step 22 to yield the corrected focused image 30, center pixel was 115, then the anti-median would be equal 

as described above. The filter system 24 includes a focused 10 100 m ™* 115 or ." 15 wh,ch 15 a ne S aUve value * P 6 

image filter 24A, which includes the application of an 20 and J he correction terms can be positive or negative, 

anti-median filter 24C at step 24A1 to the corrected focused P° sitive valu , es are used * correct P ixels w ? ich are u t0 ° dark * 

image to yield an anti-median image 30A. The image 30Ais Negative values are used to correct pixels which are too 

stored in memory for use on the following blurred image jf? ' ,. , , 

jgg The comparison function 24B2 may be written in the 

, . n 25 following manner 

FIG. 5 is a now diagram showing the blurred image .„ . , , . , . , 

processing for the filter system 24 of FIG. 3. The blurred lf sign(anU_med 1 an_blur(row >C ol)=sign(anti_rned.an_ 

image 18B is received from the sensor 10, and the correction ocus^row,co ; 

terms 26 are applied at step 22 to provide the corrected blur ^^fb^^- blu < I °^' co ! > Sr c . ,ter r than 

■ „„„ ^ At > ,.„ a fi.i f rt , r i- to,* Gito- (NUC FACTOR times (abstanti median locus 

image JOB, an output useful tor diagnostics, the niter 3Q ) ~ n ~~ ~~ 

system 24 includes a blurred filter 24B which includes the (row,col; 

application of the same anti-median filter 24C used in the taen 

focused filler 24A to the corrected blurred image. As the error__term(row,col)-anti_median_focus(row,col) 

anti-median filter is applied to the corrected blurred image, else 

the output is compared to the filter output stored from the 35 error_term(row,col)=0 

previous focus frame via a NUC comparison function 24B2, en( j if 

described more fully below. The output of the comparison where NUC FACTOR is 2 A 

function yields error terms which are used to update the ^ enw ^ afe ^ ^ ^ correction lerms 
correction terms for the next input focus image. £ach Em)r Term(roW)Col) ^ multiplied by lwo and then 
FIGS. 6, 7A-7D and 8 illustrates the application of the 40 a dded to the Correction Term(row,col) to yield updated 
anti-median filter 24C. The input image 18 is n pixels by n Correction Terms. Because the Correction Terms are of 
pixels in size. Each pixel has a numerical value representing higher precision than the image, the Correction Terms are 
brightness/intensity. For each pixel X in the image, the divided by eight before they are added to the image. The 
brightness/intensity values of X and two of its neighboring "multiply Error Terms by two, divide by eight" functions 
pixels are used to calculate a median value. The median is 4S reS ult in a one quarter (Vi) correction of the estimated error, 
calculated by sorting the three pixels in ascending order of xh e updating of the Corrections terms is illustrated diagram- 
brightness/intensity values, and selecting the middle one of maticaUy in FIG. 9, which shows the Error Terms (n 
the sorted list to be the median. Two extra rows and columns valuesxn values) being multiplied by 2, and added to the 
of alternating Maximum (M) and minimum (m) values are existing set of Correction Terms (n valuesxn values), to yield 
added to the image to handle the boundary conditions. The 50 an updated set of Correction Terms (n pixelsxn pixels), 
"anti-median" is a measure of how the pixel differs from the yh e Correction Terms are accumulated over many frames 
median. The "anti-median" is calculated by subtracting the and stored with eight times the precision of the image. Each 
pixel "X" from the median as in the following: Correction Term is divided by eight and then added to the 

antLmcdian^w, col)=mcdian( ro w ) col)-pi X ci(row > coi^oi) ^P* ™age pixel (row,col) to yield a new pixel (row,col) in 

55 the output image. This is illustrated diagrammatically in 

The result of the filtering is an nxn set 18D of anti-median FIG. 10, where the updated set of Correction Terms (n 

values for the image 18. valuesxn values) is divided by eight, and added to the input 

There are four filter patterns, illustrated as patterns 1-4 in image (n pixelsxn pixels) to yield the corrected output 

FIGS. 7A-7D, which are used in the filter system 24. In image. 

FIGS. 7A-7D, the shaded areas represents pixels that are 60 FIG. 11 illustrates an alternate embodiment of the inven- 

used in the median calculation; the center pixel is marked as tion. An alternate NUC processor 20' applies the correction 

"X." The patterns are cycled, pattern 1 through pattern 4, for terms to the input image, and employs a NUC-II filter 

each successive pair of focus/blur image frames, as illus- system 24' in the calculation of the Correction Terms. As in 

trated in FIG. 8. Thus, for a first set of focus/blur images, i.e. the embodiment of FIG. 3, the input image 18 (focused or 

images 1 and 2, filter pattern 1 is used, for the second set of 65 blurred) is received from the sensor 10. The Correction 

images 3 and 4, filter pattern 2 is used, for the third set of Terms are applied at 22 to the input image to yield a 

images 5 and 6, filter pattern 3 is used, and for the fourth set Corrected output image 30'. The Correction Terms are 
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initially zero, and are applied in same manner as described 
above regarding the processor 20. The Corrected output 
image is passed to both the Tracking system and the NUC-II 
filter system 24'. A different NUC-II filter is used for focused 
and blurred images, as will be described in further detail 5 
below. The NUC filter system 24* updates the Correction 
Terms based on the Corrected output image 30'. 

FIG. 12 illustrates the focused image processing for the 
filter system 24' of FIG. 11. The input focused image 18 A is 
received from the sensor, and the correction terms 26' are 10 
applied at step 22 to yield the corrected focused image 30', 
as described above. The filter system 24' includes a focused 
image filter 24 A', which includes the application of an 
anti-mean filter 24C at step 24AT to the corrected focused 
image to yield an anti-mean image 30 A'. The image 30A' is 15 
stored in memory for use on the following blurred image 
18B. 

FIG. 13 is a flow diagram showing the blurred image 
processing for the filter system 24' of FIG. 11. The blurred 
image 18B is received from the sensor 10, and the correction 20 
terms 26' are applied at step 22 to provide the corrected blur 
image 30B', an output useful for diagnostics. The filter 
system 24' includes a blurred filter 24B' which includes the 
application of the same anti-mean filter 24C used in the 
focused filter 24A' to the corrected blurred image. As the 25 
anti-mean filter is applied to the corrected blurred image, the 
output is compared to the filter output stored from the 
previous focus frame via a NUC-II comparison function 
24B2', described more fully below. The output of the com- 
parison function yields Error Terms which are used to update 30 
the Correction Terms for the next input focus image. 

FIGS. 14, 15A-15D and 16 illustrate the application of 
the anti-mean filter 24C\ The input image 18 is n pixels by 
n pixels in size. For each pixel X in the image, the sum of 
X and its neighboring pixels indicated by the shaded regions 35 
in the filter mask is calculated. This sum is "Alpha- 
Trimmed" by subtracting the minimum and maximum pixel 
values contained within the filter mask. The mean is calcu- 
lated by dividing the "Alpha-Trimmed" sum by number of 
pixels remaining in the sum. Filter mask pixels outside the 40 
image boundary are ignored in the calculation. This is 
illustrated by the location of the 13-pixel filter mask in FIG. 
14, which illustrates the starting position for application to 
the nxn image. Because there are no values for seven of the 
mask pixels which are outside the image boundary, they 45 
must be ignored in the calculation of the mean. Thus, for this 
starting position of the filter mask, only six pixels of the filter 
mask cover the image, two are determined to be the mini- 
mum and maximum and are "Alpha -Trimmed," and the 
remaining four pixels are averaged together to determine a 50 
mean value. As the mask is passed across the image, all 13 
mask pixels will lie within the image boundary and will be 
used in the calculation. Note that for the anti-median 
embodiment illustrated in FIG. 6, this boundary condition 
case is handled by having the extra Maximum (M) and 55 
minimum (m) values added around the outside boundary of 
the image. This cannot be done with the anti-mean filter 
because the Maximum (M) and minimum (m) values would 
bias the mean. 

The "anti-mean" is a measure of how the pixel differs $q 
from the mean. The "anti-mean" is calculated by subtracting 
the pixel "X" from the mean as in the following: 

anti_mean(row,col)-mean(row J col) — pixel(row,col) 

There are two filter patterns used for the NUC-II filter 65 
system 24', and are shown in FIGS. ISA and 15B. The 
shaded areas represent pixels that are used in the mean 



6 

calculation, with the center pixel marked as an "X". FIG. 16 
shows the cycling of the filter patterns for each successive 
pair of focus/blur frames. 

The comparison function 242B' includes the following 
steps. Each value (row,col) of the anti-mean image (focus) 
is compared to the corresponding value of the anti-mean 
image (blur) as follows: Test 1: check whether the absolute 
value of the difference between the anti-mean focus and the 
anti-mean blur is less than or equal to 5. Test 2: determine 
if the absolute value of the anti-mean focus is less than the 
absolute value of the anti-mean blur. If Test 1 and Test 2 
pass, then the error term is set to the anti-mean focus. If Test 
1 passes but Test 2 fails, then the error term is set equal to 
the anti-mean blur. If Test fails then the error term is set to 
zero and Test 2 is irrelevant. 

The Error Terms are used to update the Correction Terms 
26'. FIG. 17 diagrammatically illustrates the updating of the 
Correction Terms. Each error term (row,-col) is multiplied 
by a factor (F) based upon its magnitude and then added to 
the Correction Term (row,col) to yield updated Correction 
Terms. 

The factor (F) is calculated in the following manner. The 
absolute value of the Error Terra is used to determine the 
factor (F) by which to multiply the Error Term. Say, for 
example, that the Error Terms have an absolute value range 
of zero (0) through 100. If the absolute value of the Error 
Term is 0 or 1, the factor (F) is set to 1. If the absolute value 
of the Error Term is 2 through 7, the factor (F) is set to 2. 
If the absolute value of the Error Term is 8 through 33, the 
factor (F) is set to 4. If the absolute value of the Error Terra 
is 34 and greater, the factor (F) is set to 8. Because the 
Correction Terms are of higher precision than the image, 
they are divided by eight before they are added to the image. 
Therefore a factor (F) multiplier of 1 yields a correction of 
Va of the error, 2 is 2 /s or Va4 is Vs or Vi, and 8 is 8 /s or 1 which 
is a full correction. 

In accordance with an aspect of the invention embodied in 
the system of FIG. 11, FPN is removed from focal plane 
array imagery by comparing the output of an anti-mean filter 
that is applied to both focused and optically blurred images. 
The comparison is done in such a way that scene artifacts are 
not created and subtle scene content is not destroyed. 
Because an anti-mean filter is used, FPN is removed even in 
the presence of shading. Shading can be caused by optics, 
blue-sky, and/or dome heating. The embodiment of FIG. 3 
effectively "shuts off" in the presence of shading due to the 
use of an anti-median filter which preserves edges within the 
shading gradients. 

It is understood that the above-described embodiments are 
merely illustrative of the possible specific embodiments 
which may represent principles of the present invention. 
Other arrangements may readily be devised in accordance 
with these principles by those skilled in the art without 
departing from the scope and spirit of the invention. 

What is claimed is: 

1. An adaptive method for removing fixed pattern noise 
from focal plane array (FPA) imagery, comprising the fol- 
lowing steps: 

obtaining respective focused and blurred digitized ver- 
sions of an FPA image; 

applying a set of correction terms to the focused version 
of said FPA image to provide a corrected, focused FPA 
image; 

applying a filter to the corrected, focused version of said 
FPA image to obtain a filter focus image; 

applying said set of correction terms to said blurred 
version of the FPA image to provide a corrected, 
blurred version of said FPA image; 
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applying said filter to the corrected, blurred image to adjusting each pixel value to compensate for the dif- 

provide a filter blur image; fering sensitivity of each detector in the FPA and 

calculating fixed pattern noise errors using the filter focus provide a corrected output image, said processor corn- 
image and the filter blur image; and prising: 

employing said fixed pattern noise errors to update the 5 fi^t means for applying a set of correction terms to a 

correction terms for use in processing a next FPA focused version of said FPA image to provide a 

image. corrected, focused FPA image; 

2. The method of claim 1 wherein said filter includes an a first filter system for applying a filter to the corrected, 
anti-median filter which provides, for each pixel in an focused version of said FPA image to obtain a filter 
image, a corresponding anti-median value which is a mea- 10 focus image; 

sure of how the pixel differs from a median value of the pixel second means for applying said set of correction terms 

and a set of neighboring pixels. to said blurred version of the FPA image to provide 

3. The method of claim 2 wherein said set of neighboring a corrected, blurred version of said FPA image; 
pixels includes two pixels on opposite sides of the pixel a second filter system for applying said filter to the 
whose median value is to be calculated. corrected, blurred image to provide a filter blur 

4. The method of claim 3 wherein said step of calculating image* 

said errors includes adding boundary rows and columns of ^ me ' ans for calculati fixed ttern noise errors 

pixels values of alternating maximum and mmimum pixels ugi me ^ fQCUS . aQd me fiUer Mur ; 

values to said corrected output image to bound said output and 

imageand accommodate boundary conditions 20 fourth means for employing said fixed pattern noise 

5. The method of claim 2 wherein said step of calculating erfors , 0 date , he corrcction t6rms for use in 
said errors includes comparing corresponding values of said processing a next FPA image 

focus and blur images, and calculating a corresponding error n ^ system of daim 1Q wherein ^ fiher inc , udes afl 

term in dependence on said comparison anti-median filter which provides, for each pixel in an 

6. The method of claim 1 wherein said filter includes an ^ a corresponding anti . median valu6 which is a mea . 
anti-mean filler which provides, for each pixel in an image, sure of how the pixel differs from a median value of the pixel 
a corresponding anti-mean value which is a measure of how and a se , of neighboring pixels . 

the p,xel differs from a mean value of the pixel and a set of u ^ sys(em of daim n wherein ^ M of neighbor . 

neighboring pixels. . . ,„ ing pixels includes two pixels on opposite sides of the pixel 

7. The method of claim 6 wherein said set of neighboring whos6 median vahl6 ^ t0 bc ^huhlcd. 

pixels includes a group of neighboring pixels, and wherein 13 ^ (em of daim n wherein said , hird means fo , 

pixels in said group having maximum and minimum values calculati ^ errors inchldes means for comparing ^ 

are excluded from calculation of said mean value d; values of ^ focus and bbr ^ and m6ans 

8 The method of claim 6 wherein said step of calculating for calculaling a coa^d^ error term in dependence on 

said errors includes comparing corresponding values of said sa j d comMr j son 

focus and blur images, and calculating a corresponding error u ^ system of claim 10 wherein said fi „ er indudcs an 

term in dependence or. said comparison anti-mean filter which provides, for each pixel in an image, 

9. The method of claim 1 wherein said FPA image is an , an ti-mean value which is a measure of how 
infrared image. . 40 the pixel differs from a mean value of the pixel and a set of 

10. An adaptive sensor and non-uniformity compensation neighboring pixels 

system for providing a corrected output image from a focal 15 ^ ffl of daiffl u wherein ^ &c{ of nci hbor . 

plane array (FPA) wherein fixed pattern noise has been ^ ixds indudes a of neighboring pixcls> and 

compensated m FPA imagery, comprising: whefein pixds m said ^ faaving maximum and mini . 
a sensor comprising sensor optics, a focal plane array of 45 mum values are excluded from calculation of said mean 

infrared detectors for producing FPA images, and value. 

analog-to-digital conversion apparatus for converting 16, The system of claim 13 wherein said third means for 

said FPA images into digitized FPA images comprising calculating said errors includes means for comparing corre- 

a set of pixel values, the sensor optics adapted to sponding values of said focus and blur images, and means 
provide focused and blurred versions of images; 5Q f or calculating a corresponding error term in dependence on 

an adaptive non-uniformity compensation processor for said comparison, 
compensating for non-uniform sensitivities of said 

infrared detectors to infrared radiation, said processor * * * * * 
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